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FOREWORD 


The  U.S.  Army  has  made  a  substantial  commitment  to  the  use  of  networked  simulations 
for  training,  readiness,  concept  development,  and  test  and  evaluation.  Augmented  Reality  (AR) 
technology,  which  has  the  capability  to  supplement  information  received  directly  from  the  actual 
environment  with  computer-generated  information,  has  the  potential  to  enhance  both  the  conduct 
of  military  operation  and  training  for  those  operations.  However,  the  effectiveness  of  AR 
technology  for  either  purpose  has  not  been  established. 

This  report  describes  an  experiment  in  an  ongoing  program  of  research  addressing  the  use 
of  virtual  environment  and  AR  technology  for  training  dismounted  Soldiers.  The  findings  from 
this  research  can  be  used  to  recommend  AR  characteristics  and  application  methods  that  should 
be  incorporated  into  future  AR  training  and  operational  systems. 

The  U.S.  Army  Research  Institute,  Simulator  Systems  Research  Unit,  conducts  research 
with  the  goal  of  providing  information  that  will  improve  the  effectiveness  and  efficiency  of 
training  simulators  and  simulations.  The  work  described  here  is  a  part  of  ARI  Research  Task 
233,  VICTOR:  Virtual  Individual  and  Collective  Training  for  Future  Warriors 

MICHELE  SAMS,  Ph.D. 

Director 
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THE  PERCEPTION  AND  ESTIMATION  OF  EGOCENTRIC  DISTANCE  IN  REAL  AND 
AUGMENTED  REALITY  ENVIRONMENTS 

EXECUTIVE  SUMMARY 


Research  Requirement: 

Previous  research  using  verbal  judgments  of  distance  have  shown  that  the  distances  tend 
to  be  underestimated.  The  extent  to  which  distances  are  underestimated  is  greater  with  virtual 
environments  than  with  real  world  environments.  A  primary  focus  of  this  study  was  to 
determine  whether  the  judged  distances  to  virtual  objects  in  a  real  world  environment  produces 
results  similar  to  those  found  previously  for  virtual  objects  in  a  virtual  world. 

Procedure: 

Participants  judged  distances  to  a  single  object  during  two  sets  of  twelve  test  trials.  The 
objects  were  placed  at  varying  distances.  During  one  set,  participants  judged  the  distance  to  the 
real  object  while  in  the  other  set  they  judged  the  distance  to  a  virtual  representation  of  that 
object.  Half  of  the  participants  provided  their  response  verbally  with  a  distance  estimate  in  their 
unit  of  choice  (i.e.,  feet  or  meters),  and  the  other  half  provided  their  response  by  moving  a 
remote  controlled  robot  to  the  location  of  the  previously  viewed  object. 

Findings: 

Participants  who  indicated  their  distance  judgments  to  an  object’s  perceived  location  by 
moving  the  robot  were  significantly  more  accurate  than  those  who  gave  verbal  estimates  of  the 
distances.  Participants  were  significantly  more  accurate  at  judging  the  distance  to  real  objects  in 
a  real  environment  than  they  were  to  virtual  objects  in  an  augmented  environment.  However, 
this  latter  effect  was  entirely  due  to  females  judging  virtual  objects  much  less  accurately  than 
they  judged  real  objects.  Males  judged  virtual  objects  as  well  as  they  judged  real  objects. 

Utilization  and  Dissemination  of  Findings: 

These  results  lend  further  support  to  the  notion  that  estimated  distance  does  not 
accurately  represent  perceived  distance.  Unless  the  task  being  performed  specifically  requires  a 
numerical  estimate  of  distance,  it  is  recommended  that  methods  similar  to  our  distance 
replication  method  be  used  to  accurately  determine  perceived  distance. 
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THE  PERCEPTION  AND  ESTIMATION  OF  EGOCENTRIC  DISTANCE 
IN  REAL  AND  AUGMENTED  REALITY  ENVIRONMENTS 


INTRODUCTION 

Modem  training  programs  incorporate  technologies  that  have  the  potential  to  improve  the 
effectiveness  of  training  as  well  as  increase  safety.  Virtual  and  augmented  reality  (AR)  are  two 
similar  technologies  that  have  shown  great  promise  to  improve  training;  however,  computer 
generated  stimuli  might  be  perceived  differently  than  real  world  stimuli.  The  goal  of  this 
research  is  to  investigate  human  perception  and  performance  in  an  environment  in  which  the  real 
world  environment  is  augmented  with  virtual  objects.  The  real  world  is  modeled  and  tracked  in 
relation  to  the  user’s  head  and  body  position  so  that  the  virtual  objects  may  be  overlaid  onto  the 
real  world  in  their  correct  and  desired  positions.  This  is  generally  referred  to  as  AR  and  the 
environment  may  be  referred  to  as  an  augmented  or  mixed  reality  environment.  AR  is  a  new 
technology  and  as  such,  much  of  the  early  work  in  this  area  has  concentrated  on  describing 
potential  AR  applications  and  overcoming  the  technical  challenges  such  as  accurate  registration 
and  tracking  (Azuma,  1997).  Researchers  have  only  recently  begun  to  investigate  human 
perception  and  performance  in  augmented  and  mixed  reality  environments  (Livingston,  et  al., 
2004;  Tang,  Owen,  Biocca  &  Mou,  2003;  Azuma  &  Furmanski,  2003;  Kirkley,  2003).  These 
empirical  studies  are  providing  data  on  human  interactions  with  AR  technology  that  are  essential 
for  determining  the  usefulness  of  current  AR  for  training  and  performance  enhancement. 

The  Augmented  Reality  Performance  Assessment  Battery  (ARPAB)  was  designed  as  a 
test  battery  to  explore  perception  and  performance  in  augmented  environments.  Previously, 
Kirkley  (2003)  had  used  the  distance  estimation  task  from  this  battery  to  explore  distance 
estimation  in  an  AR  environment;  results  showed  that  participants’  distance  judgments  were 
more  accurate  when  the  room  was  visible,  when  the  object  was  on  the  ground  vs.  floating,  and 
when  the  object  was  real  vs.  virtual.  Distance  estimation  is  important  because  many  tasks 
require  an  individual  to  be  able  to  perceive  the  location  of  an  object  accurately  enough  to  be  able 
to  interact  with  it.  Kirkley’s  work  was  unique  by  using  AR  to  deliver  real  and  virtual  objects  in  a 
real  environment,  and  interesting  since  it  found  significant  results  describing  limitations  of  AR 
with  distance  presentation,  but  some  shortcomings  should  be  noted.  For  example,  the  head 
tracking  system  used  by  Kirkley  produced  significant  drift  in  the  virtual  objects,  which  could 
have  adversely  affected  his  results.  Kirkley’s  participants  only  viewed  the  objects  with  one  eye, 
which  possibly  could  also  influence  his  results.  In  addition,  Kirkley’s  tasks  required  judging 
four  widely  separated  distances  repeatedly  under  different  conditions.  Participants  tested  under 
various  conditions  could  easily  recall  their  previous  estimates  at  each  of  the  four  distances  and 
repeat  these  estimates  regardless  of  the  experimental  condition  presented.  This  could  have 
decreased  variability  and  produced  better  results  for  the  AR  condition  than  would  have  been 
obtained  had  more  distances  been  used.  Due  to  these  issues  we  wanted  to  re-evaluate  distance 
estimation  in  an  AR  environment  using  more  discrete  distances  with  smaller  intervals  between 
the  distances,  a  better  head  tracker,  and  both  eyes  to  make  the  distance  judgments. 

Distance  Estimation  -  Verbal 

In  general,  people  tend  to  underestimate  egocentric  distance  judgments,  i.e.,  they  perceive 
and  report  that  the  distance  from  them  to  some  other  object  is  shorter  than  what  the  distance  truly 


is.  Studies  consistently  support  this  in  real  environments,  especially  when  the  egocentric 
distances  are  reported  verbally  (Gihnsky,  1951;  Harway,  1963). 

These  difficulties  in  distance  judgments  generalize  to  immersive  virtual  environments  as 
well.  While  immersed  in  a  virtual  environment,  distance  judgments  from  the  location  of  the 
person’s  avatar  to  a  virtual  object  are  significantly  underestimated  when  reported  verbally 
(Witmer  &  Kline,  1 998;  Witmer  &  Sadowski,  1 998;  Henry  &  Furness,  1 997;  James  &  Caird, 
1995;  Lampton,  Singer,  McDonald,  &  Bliss,  1995).  Are  these  distance  judgment  difficulties 
errors  in  the  actual  perception  of  the  object’s  spatial  location,  or  are  they  errors  in  the  verbal 
reporting?  More  specifically,  could  people  accurately  interact  with  the  object  regardless  of  the 
inaccuracy  of  the  number  of  feet  or  meters  they  assign  to  the  distance? 

Distance  Estimation  -  Replication 

Besides  giving  a  verbal  estimate  of  distance  judgments,  people  can  provide  a  distance 
judgment  by  replicating  the  distance  in  some  way.  Research  shows  that  people  can  interact  with 
the  virtual  objects  at  varying  distances  when  tasks  such  as  blindwalking  are  used  (Thomson, 
1983;  Reiser  et  al.,  1990;  Loomis  et  ah,  1992).  Blindwalking  involves  the  person  closing  their 
eyes  after  viewing  a  virtual  object  or  location,  and  walking  the  distance  they  perceived  to  that 
spatial  location.  This  direct  interaction  circumvents  the  inaccuracies  of  verbal  distance 
judgments;  however,  the  improvements  made  over  verbal  judgments  might  be  mitigated  when 
comparing  both  types  of  estimation  methods  in  virtual  environments.  For  example,  other  studies 
focusing  on  the  importance  of  the  environment  type  on  the  distance  judgments  show  how 
distance  estimation  using  either  verbal  judgments  or  direct  interaction  tasks  is  considerably 
shorter  in  the  virtual  world  than  in  the  real  world  (Lampton,  Singer,  McDonald,  &  Bliss,  1995; 
Witmer  and  Kline,  1998;  Witmer  and  Sadowski,  1998). 

The  Present  Experiment  and  Hypotheses 

AR  shows  great  promise  as  a  training  tool  as  well  as  a  technology  to  aid  task 
performance.  There  is  much  we  need  to  learn  about  the  effectiveness  of  AR.  Not  only  do  we 
need  to  know  when  AR  can  reasonably  serve  as  a  substitute  for  real  environments,  but  the  extent 
to  which  it  can  improve  on  what  we  can  do  in  real  environments,  both  as  a  performance  enhancer 
and  a  training  tool.  AR  research,  especially  research  that  addresses  human  performance  in 
augmented  environments  is  in  its  nascent  period.  There  is  much  to  be  done  and  only  a  few 
research  labs  that  have  the  necessary  equipment  and  expertise  to  perform  the  work.  A  primary 
objective  of  this  research  was  to  compare  the  accuracy  of  distance  judgments  to  virtual  objects 
that  augment  a  real  environment  (AR)  with  judgments  to  real  objects  in  a  real  environment.  A 
second  major  goal  of  this  research  was  to  determine  if  accuracy  of  distance  judgments  could  be 
improved  by  allowing  participants  to  indicate  the  distance  to  an  object  by  maneuvering  a  robot  to 
the  perceived  location  of  the  object. 

Previous  research  using  verbal  judgments  of  distance  have  shown  the  distances  tend  to  be 
underestimated.  The  extent  to  which  distances  are  underestimated  is  greater  with  virtual 
environments  than  with  real  world  environments  (Lampton,  Singer,  McDonald,  &  Bliss,  1995; 
Witmer  &  Kline,  1998).  A  primary  focus  of  this  experiment  was  to  determine  whether  the 
judged  distances  to  virtual  objects  in  a  real  world  environment  produces  results  similar  to  those 
found  previously  for  virtual  objects  in  a  virtual  world.  In  this  paper,  we  will  describe  the 
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perceptual  and  performance  issues  that  were  identified  during  research  that  utilized  the  ARPAB 
distance  estimation  test.  These  tests  are  relevant  to  potential  applications  of  AR  technology  and 
will  be  used  to  identify  and  describe  any  perceptual  and/or  cognitive  issues  associated  with 
judging  distance  to  virtual  objects  that  are  overlaid  onto  a  real  world  environment. 


EXPERIMENTAL  METHOD 

The  participants  were  32  undergraduate  and  graduate  students  from  the  University  of 
Central  Florida.  There  were  18  females  and  14  males,  ranging  in  age  from  18  to  51  years  ( M  = 
24.66,  SD  =  8.1 ).  All  participants  had  unaided  or  corrected  20/20  vision  and  had  normal  color 
vision.  The  participants  were  assigned  randomly  to  the  experimental  conditions. 

Experimental  Design 

The  experiment  used  a2x2x2x2x2x  12  mixed  model  repeated  measures  design.  The 
between-subject  variables  were  object  type  (common  or  abstract  objects),  object  size  (small  or 
large),  environment  order  (augmented  first  or  real  first)  and  judgment  method  (verbal  or 
replication).  The  within  subjects  variables  included  environment  type  (real  and  augmented)  and 
distance  (12  levels).  Two  random  presentation  orders  for  object  distance  were  used,  so  that 
distances  were  neither  presented  in  ascending  or  descending  order.  To  control  for  gender  effects 
or  effects  resulting  from  distance  presentation  order,  gender  and  distance  order  were  included  as 
covariates. 


Materials 

The  equipment  used  in  this  research  included  hardware  and  software  for  presenting 
virtual  objects  in  an  AR  environment,  a  remotely  controlled  robot  with  the  hardware  and 
software  for  controlling  it,  and  three  questionnaires.  Additional  equipment  included  real  objects 
and  their  virtual  representations  (Virtual  Reality  Markup  Language  [VRML]  objects),  a  string  of 
lights  for  calibration,  a  Strait-line  laser  tape  measure  for  measuring  distance  to  the  robot,  and 
eight  500- watt  halogen  spotlights  for  illuminating  the  real  world  objects.  The  objects  included  a 
soldier  mannequin  (167.6  x  50.8  x  21 .6  cm),  a  computer  monitor  (43.2  x  41.9  x  42.5  cm),  an 
adjustable  office  chair  (105.4  x  59  x  50.8  cm),  an  office  table  (74.9  x  121.9  x  91.4  cm),  a  milk 
jug  (25.4  x  15.2  x  15.2  cm),  a  small  rectangular  box  (29.2  x  22.9  x  19  cm),  a  large  rectangular 
box  (144.8  x  68.6  x  50.8  cm),  a  small  sphere  (35.6  x  35.6  x  35.6  cm),  and  a  large  sphere  (66  x  66 
x  66  cm).  The  VRML  virtual  objects  were  displayed  in  the  real  environment  in  the  same  position 
and  distance  as  the  real  objects  and  were  designed  to  be  perceived  as  approximately  the  same 
size  as  their  real  world  counterparts  (see  Figures  1  and  2). 
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Figure  1 


Figure  2 


View  through  HMD  of  real  (left  column)  &  virtual  familiar  objects  (right  column). 


View  through  HMD  of  real  (left  column)  &  virtual  abstract  objects  (right  column). 
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AR  hardware  and  software.  A  version  of  the  Battlefield  Augmented  Reality  System 
(BARS  )  was  used  in  conjunction  with  ARPAB  software  to  present  virtual  objects  superimposed 
on  the  see-thru  view  of  the  real  world  environment.  The  BARS  hardware  included  a  laptop 
computer  affixed  to  the  upper  torso  of  a  mannequin  and  mounted  on  a  cart  for  ease  of  use,  a 
Sony  Glasstron  LDI-D100B  stereo  optical  see-thru  display  for  displaying  high  resolution  virtual 
3-D  color  objects,  and  the  Intersense-900  tracking  system  for  tracking  user  position  and 
orientation  (see  Figure  3).  The  Glasstron  display  provided  a  20-degree  horizontal  field-of-view 
of  the  virtual  computer  generated  scene  in  each  eye.  The  ARPAB  software  permitted  the 
selection,  ordering,  and  timing  of  object  presentation,  as  well  as  the  ability  to  record  the  distance 
judgments  for  each  participant.  The  ARPAB  software  resided  on  Windows  PC  that  was  used  to 
control  the  presentation  of  the  experimental  trials  and  capture  the  data. 


Figure  3.  Battlefield  augmented  reality  system  (BARS)  and  Sony  Glasstron  HMD. 


Remotely  controlled  robot  hardware  and  software.  A  small  remote-controlled  vehicular 
style  robot  (see  Figure  4)  was  used  to  allow  participants  to  replicate  the  perceived  distances  to 
real  and  virtual  target  objects  by  moving  the  vehicle  along  a  lighted  corridor  to  the  perceived 
object  location.  The  robot  was  controlled  remotely  using  an  antenna  and  joystick  control  box.  A 
second  laptop  was  used  for  initializing  the  robot  software,  monitoring  the  robot  power  and  other 
critical  functions,  as  well  as  communicating  direction  and  control. 
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Figure  4.  Controller  and  robot  shown  moving  along  the  illuminated  corridor. 


Questionnaires.  Three  questionnaires  were  used  -  one  for  measuring  presence,  another 
for  measuring  comfort  (simulator  sickness),  and  a  third  for  collecting  participant  demographic 
information  and  for  identifying  prior  relevant  experiences.  The  Presence  Questionnaire  (PQ) 
Version  4.0  (Witmer,  Jerome,  &  Singer,  2005)  included  33  items  to  determine  the  degree  to 
which  the  participants  reported  a  sense  of  presence  in  the  AR  Environment.  The  second 
questionnaire  was  a  16-item  Simulator  Sickness  Questionnaire  (SSQ)  (Kennedy,  Lane,  Berbaum, 
&  Lillienthal,  1993)  to  assess  the  amount  of  discomfort  produced  by  the  AR  environment.  The 
demographic  questionnaire  also  contained  16  items. 

Experimental  Environment.  The  experiment  was  conducted  in  the  SFC  Paul  Ray  Smith 
Simulation  and  Training  Technology  Center  in  the  Central  Florida  Research  Park.  The  testing 
environment  consisted  of  a  long  corridor  that  included  numerous  real  world  cues  and  landmarks. 
The  testing  area  was  artificially  illuminated  by  overhead  fluorescent  lighting  that  provided 
adequate  illumination  for  reading  instructions  or  for  easily  viewing  objects  with  the  naked  eye. 
Viewing  real  objects  through  the  Glasstron,  however,  was  much  more  difficult,  especially  at 
longer  distances,  and  necessitated  the  use  of  spotlights.  As  much  as  possible,  the  spotlights  were 
placed  out  of  view  along  the  edge  of  the  corridor  and  strategically  positioned  to  illuminate  the 
real  world  objects.  The  spacing  between  adjacent  spotlights  was  variable  and  typically  the  light 
provided  by  the  lamp  could  be  seen,  but  not  the  lamp  itself.  Because  the  lamps  did  not  uniformly 
light  the  corridor,  the  spotlights  from  the  lamps  provided  additional  landmarks  that  participants 
could  use,  along  with  other  landmarks  in  the  environment,  to  remember  object  location.  The 
spotlights  were  present  when  either  real  or  virtual  objects  were  presented. 
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Procedure 

After  consenting  to  participate,  participants  completed  the  demographic  questionnaire 
and  a  baseline  SSQ.  They  then  entered  the  research  laboratory  where  they  read  instructions 
describing  the  experimental  task  and  ARPAB  calibration  procedures.  Standing  at  the  zero 
distance  position,  represented  by  crossmarks  on  the  floor,  participants  placed  the  Glasstron  on 
their  head  and  positioned  it  for  comfortable  viewing.  Participants  also  strapped  the  carrying  case 
containing  the  Glasstron  battery  pack  over  their  shoulder.  The  Glasstron  was  set  for  minimum 
opacity  to  yield  the  best  view  of  the  real  environment.  This  opacity  setting  also  provided  a  clear 
view  of  the  virtual  objects. 

The  participant’s  first  task  was  to  calibrate  the  ARPAB  software  so  that  the  virtual 
objects  were  displayed  in  their  correct  locations.  This  was  accomplished  by  first  making  the 
computer  wireframe  representation  of  the  environment  visible  and  having  participants  center  the 
wireframe  in  their  field-of-view  to  lock  their  view  to  the  wireframe  position,  and  then  by  asking 
participants  to  move  their  head  to  align  the  left  back  vertical  edge  of  the  wireframe  with  a  string 
of  lights  placed  in  the  corresponding  real  world  position.  Just  prior  to  calibration,  one  researcher 
showed  participants  photographs  of  the  wireframe  correctly  overlaid  on  the  real  world  and 
explained  the  procedure  in  detail  to  ensure  that  the  participants  clearly  understood  the  calibration 
procedures  (see  Appendix  A).  Because  the  calibration  procedure  was  neither  quick  nor  easy, 
participants  were  asked  not  to  adjust  the  position  of  the  Glasstron  following  calibration  to  avoid 
the  need  to  recalibrate.  After  the  calibration  was  completed,  the  wireframe  was  made  invisible. 

Following  calibration,  the  participants  received  6  practice  trials  to  familiarize  them  with 
the  experimental  procedures  and  provide  practice  in  judging  distance  to  real  and  virtual  objects. 
These  trials  also  provided  practice  in  controlling  the  movement  of  the  robot  for  the  distance 
replication  group.  A  real  object  (a  table  121.9  x  91.4  x  76.2  cm)  was  shown  for  three  of  the 
practice  trials,  while  a  3-D  virtual  representation  of  the  table  was  shown  for  the  other  three 
practice  trials.  The  table  was  presented  at  distances  of  1.5,  3.7,  5.8,  7.9,  10.1,  12.2  meters. 

During  practice  the  participants  did  not  receive  any  feedback  on  the  accuracy  of  their  distance 
judgments. 

After  completing  the  practice  trials  participants  judged  distances  to  a  single  object  during 
two  sets  of  twelve  test  trials.  The  objects  were  placed  at  distances  of  1.5,  3.7,  5.8,  7.9,  10.1, 

12.2,  14.3,  16.5,  18.6,  20.7,  22.9,  and  25  meters.  During  one  set,  participants  judged  the  distance 
to  the  real  object  while  in  the  other  set  they  judged  the  distance  to  a  virtual  representation  of  that 
object.  Half  of  the  participants  judged  the  distance  to  the  virtual  object  first  while  the  other  half 
judged  the  distance  to  the  real  object  first. 

The  object  presented  to  the  participant  was  randomly  selected  from  a  set  of  8  objects. 

Four  of  the  objects  were  of  relatively  small  size  and  four  were  substantially  larger.  Half  of  the 
objects  were  3-D  abstract  shapes  that  can  vary  widely  in  size  (large  or  small  sphere,  large  or 
small  rectangle)  and  half  of  the  objects  were  familiar  objects  where  the  approximate  size  was 
known  (adjustable  office  chair,  life-size  soldier  mannequin,  computer  monitor,  milk  jug).  This 
was  done  because  knowledge  of  object  size  can  potentially  provide  a  cue  to  the  object  distance. 
Half  of  the  participants  received  distance  Order  1,  while  the  other  half  received  distance  Order  2. 

On  each  trial  the  following  sequence  of  events  occurred.  Participants  began  each  trial 
with  their  eyes  closed.  When  the  object  was  presented,  participants  opened  their  eyes  and 
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viewed  the  objeet  for  12  seeonds.  Participants  then  elosed  their  eyes  and  the  objeet  was 
removed.  Participants  opened  their  eyes  and  rendered  their  distance  judgment.  Participants  in 
the  estimation  group  verbally  reported  the  distance  in  their  preferred  units  (feet  or  meters),  while 
participants  in  the  replication  group  maneuvered  a  robot  to  the  location  where  they  judged  the 
previously  displayed  object  to  have  been  positioned.  Participants  closed  their  eyes  while  their 
distance  judgment  was  recorded.  The  object  was  then  repositioned  in  preparation  for  the  next 
trial.  For  the  replication  group,  the  robot  was  returned  to  a  position  just  forward  of  the 
participant.  Participants  opened  their  eyes  after  this  repositioning  to  make  their  next  judgment. 

After  completing  their  last  trial,  participants  completed  another  SSQ  and  the  PQ.  They 
were  then  assigned  pay  or  elass  eredit  for  their  participation  and  received  a  short  written  debrief. 

RESULTS 

A  primary  objective  of  this  research  was  to  eompare  the  accuracy  of  distance  judgments 
to  virtual  objects  that  augment  a  real  environment  (augmented  reality)  with  judgments  to  real 
objeets  in  a  real  environment.  A  seeond  major  goal  of  this  research  was  to  determine  if  aeeuraey 
of  distance  judgments  eould  be  improved  by  allowing  participants  to  indicate  the  distance  to  an 
objeet  by  maneuvering  a  robot  to  the  pereeived  location  of  the  objeet. 

We  selected  signed  error  as  our  basie  index  of  the  aeeuraey  of  distance  judgments. 

Signed  error  is  defined  as  the  judged  distance  minus  the  true  distance  to  the  target.  Beeause  the 
error  in  distance  judgments  is  often  proportional  to  the  distance  of  the  target  from  the  observer 
( Witmer  &  Kline,  1 998),  we  ehose  to  report  signed  relative  error  rather  than  the  more  basie 
signed  error.  Signed  relative  error  is  defined  as  the  signed  error  divided  by  the  true  target 
distance.  Figure  5  summarizes  the  distance  judgment  results  for  the  two  environment  types 
(augmented  and  real)  and  two  judgment  types  (estimation  and  replication).  For  exposition 
purposes  signed  relative  error  is  shown  in  Figure  5  as  a  percentage  of  the  true  target  distance. 

The  figure  indicates  decreased  performance  when  the  verbal  estimation  method  was  used  and 
when  the  real  environment  was  augmented  with  virtual  objeets 


Augmented  Real  Augmented  Real 

Estimation  Estimation  Replication  Replication 


Figure  5.  Distance  judgment  aeeuraey  by  environment  type  and  judgment  type. 
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The  results  illustrated  in  Figure  5  are  supported  by  the  mixed  model  repeated  measures 
analysis.  Participants  who  indicated  their  distance  judgments  to  an  object’s  perceived  location  by 
moving  the  robot  were  significantly  more  accurate  than  those  who  gave  verbal  estimates  of  the 
distances,  F  ( 1 , 1 4)  =  25.35,  p  <  .001 .  Participants  were  significantly  more  accurate  at  judging 
the  distance  to  real  objects  in  a  real  environments  than  they  were  to  virtual  objects  in  an 
augmented  environment,  F  (1,14)  =  33.85,/?  <  .001.  However,  this  latter  effect  was  entirely  due 
to  females  judging  virtual  objects  much  less  accurately  than  they  judged  real  objects.  Males 
judged  virtual  objects  as  well  as  they  judged  real  objects.  The  significant  gender  by  environment 
type  interaction,  /r(l,14)  =  18.35  ,  p  <  .001,  is  shown  in  Figure  6. 


Gender 
J  Females 
J  Males 


Figure  6.  Differential  gender  effects  on  distance  judgment  accuracy  to  real  and  virtual  objects. 


No  other  main  effects,  including  object  distance  had  significant  effects  on  signed  relative 
error.  However  the  interaction  between  distance  and  distance  order  was  significant,  F{  1,14)  = 
8.4,  p  <  .05,  suggesting  that  distance  at  which  the  object  was  viewed  significantly  affected 
relative  error  for  distance  order  1,  but  not  for  distance  order  2.  The  3-way  interaction  between 
distance  by  distance  order  by  object  size  was  significant  as  well,  F  (1,  14)  =  9.12,  p  <  .05. 
Distance  order  also  interacted  with  the  environment  type,  having  more  effect  on  virtual  objects  in 
an  augmented  reality  environment  than  on  real  objects  in  a  real  environment,  F(l,14)  =  15.03,/? 
<  .01 .  None  of  the  interactions  with  distance  order  were  of  particular  interest  and  it  is  likely  that 
some  of  these  effects  are  spurious. 

Environment  order  interacted  significantly  with  environment  type,  F(l,14)  =  4.89,/?  < 
.05.  The  amount  of  error  in  judging  the  distance  to  virtual  objects  was  significantly  reduced 
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when  preceded  by  distance  judgments  of  real  objects  (see  Figure  4).  Hence  previous  experience 
in  judging  distance  to  the  real  objects  improves  subsequent  judgments  of  the  virtual 
representations  of  those  objects.  There  were  two  significant  3-way  interactions  involving 
environment  type.  The  environment  type  by  environment  order  by  object  size  interaction  was 
significant,  F(  1,  14)  =  8.74,  p  <  .01.  The  environment  type  by  judgment  type  by  object  type 
interaction  was  also  significant,  F  (1,14)  =  8.219,/?  <  .05.  Except  for  the  real-world 
representations  of  the  abstract  objects,  participants  always  were  more  accurate  for  real  objects 
than  for  virtual  objects  in  an  augmented  environment.  There  was  also  a  significant  four-way 
interaction,  environment  type  by  environment  order  by  object  type  by  judgment  type  that  could 
not  be  interpreted. 


Environment 

Order 

J  Virtual  First 
5  Real  First 


Real/Real  Augmented/Virtual 

Environment  Type/Object 


0.45H 


Figure  7.  Presentation  order  effects  on  distance  judgment  accuracy. 


As  this  was  our  first  research  project  using  AR,  we  administered  the  presence  and  SSQ  to 
assess  presence  and  simulator  sickness  in  an  AR  environment.  The  PQ  results  suggest  that 
participants  experienced  significant  presence  (M=  120.62,  SD  =  28.33).  Responses  to  individual 
PQ  items  suggest  that  the  participants  adjusted  quickly  to  the  augmented  environment  and 
reported  that  the  visual  and  control  interfaces  did  not  interfere  with  task  performance. 
Surprisingly,  the  SSQ  simulator  sickness  scores  were  slightly  high  (M  =  31 .44,  SD  =  39.26).  We 
did  not  anticipate  much  sickness  because  the  participants  were  stationary  with  little  head 
movement  and  the  objects  being  judged  also  did  not  move.  Despite  this,  one  participant  (SSQ  = 
104.72)  reported  nausea  and  a  slight  headache  (both  symptoms  of  simulator  sickness).  This 
participant  also  reported  a  history  of  migraine  headaches  with  tunnel  vision  and  dizziness.  The 
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higher  scores  were  most  likely  due  to  the  extended  exposure  time  to  the  AR  environment.  The 
sessions  lasted  approximately  45  minutes. 

DISCUSSION 

The  significant  main  effect  of  environment  type  in  this  research  indicates  that  participants 
underestimate  the  distance  to  virtual  objects  in  a  real  environment  more  than  they  do  for  real 
objects  in  a  real  environment.  However,  further  analyses  of  the  data  revealed  that  this  was  only 
true  for  our  female  participants.  It  is  not  clear  what  is  responsible  for  this  apparent  gender- 
specific  effect.  Differences  in  spatial  ability  between  our  male  and  female  participants  is  a 
possible  explanation,  but  why  would  spatial  ability  differences  affect  judgments  to  virtual  objects 
to  a  greater  degree  than  it  affects  judgments  to  real  objects?  Perhaps,  judging  the  position  of 
virtual  objects  in  the  real  world  may  require  superior  spatial  visualization  skills,  since  the  virtual 
objects  are  less  likely  than  real  objects  to  appear  to  be  clearly  anchored  to  a  position  on  the  floor. 
Hence  participants  rely  on  visualization  skills  to  accurately  determine  the  position  where  a 
virtual  object  intersects  the  floor.  Studies  examining  spatial  visualization  skills  in  men  and 
women  have  generally  shown  men  to  have  better  visualization  skills  (Eisenberg  &  McGinty, 
1977;  McGee,  1979;  Linn  &  Petersen,  1986;  Battista,  1990).  A  difference  in  visualization  skills 
between  our  men  and  women  participants,  assuming  they  exist,  might  explain  our  otherwise 
puzzling  results. 

The  method  used  to  obtain  distance  judgments  can  have  a  dramatic  effect  on  the  accuracy 
of  those  judgments  (Thomson,  1983;  Elliot,  1987;  Rieser,  Ashmead,  Talor,  &  Youngquist,  1990; 
Witmer  &  Sadowski,  1998).  In  the  current  experiment,  asking  participants  to  replicate  the 
perceived  distance  by  moving  a  robot  to  the  perceived  location  produced  more  accurate  distance 
judgments  than  did  the  verbal  distance  estimation  procedure.  This  finding  suggests  that  the 
verbal  estimates  of  distance  by  untrained  participants  may  not  accurately  reflect  perceived 
distances.  The  accuracy  of  distance  judgments  is  clearly  improved  by  using  the  distance 
replication  procedure  to  obtain  the  judgments  (see  Figure  5).  Furthermore,  the  results  shown  in 
Figure  5  suggest  that  the  effects  of  distance  judgment  method  were  greater  than  were  the  effects 
of  environment  type.  The  magnitude  of  the  distance  judgment  error  was  considerably  larger  for 
the  estimation  condition  in  the  real  environment  than  it  was  for  the  replication  condition  in  the 
augmented  environment.  These  results  lend  further  support  to  the  notion  that  estimated  distance 
does  not  accurately  represent  perceived  distance.  Unless  the  task  being  performed  specifically 
requires  a  numerical  estimate  of  distance,  it  is  recommended  that  methods  similar  to  our  distance 
replication  method  be  used  to  accurately  determine  perceived  distance. 
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APPENDIX  A:  CALIBRATION  INSTRUCTIONS 


2. 


Focus  on  the  virtual 
wireframe  of  the  hallway, 
specifically  try  to  keep  in 
the  center  of  the  display 
the  location  where  all  the 
virtual  walls  converge 

When  you  have  the 
wireframe  centered,  please 
inform  the  experimenter 
so  he  can  lock  the 
wireframe  in  place. 


Notice  the  real  wo  rid 
walls  are  not  aligned  with 
the  wireframe. 


5. 


Once  the  wireframe  is 
locked  inthe  center  ofthe 
display,  you  may  move 
your  head  to  align  the  real 
world  walls  with  the 
wireframe  walls  Please 
line  up  the  very  end  ofthe 
wall  on  the  left  hand  side 
with  the  lights  hanging 
half  way  down  the  hall 
When  you  have  aligned 
them  as  best  you  can, 
inform  the  experimenter 
so  he  can  lock  the 
wireframe  walls  onto  the 
real  world  walls 


t.* 


A-l 


A-2 


APPENDIX  B:  SIMULATOR  SICKNESS  QUESTIONNAIRE 


Simulator  Sickness  Questionnaire  (SSQ) 

Kennedy,  Lane,  Berbaum,  and  Lillenthal  (1993) 

Keyboard  Directions:  Position  the  cursor  over  the  response  that  you  want  to  select  for  a  given 
question,  then  click  the  left  mouse  button  to  select  it.  Use  the  scroll  bar  or  PgDn  button  to  move 
to  the  next  set  of  (off-screen)  questions.  Please  tell  the  experimenter  when  you  are  finished. 

Instructions:  Please  indicate  the  severity  of  symptoms  that  apply  to  you _ 


1 .  General  Discomfort 

None 

Slight 

Moderate 

Severe 

2.  Fatigue 

None 

Slight 

Moderate 

Severe 

3.  Headache 

None 

Slight 

Moderate 

Severe 

4.  Eye  Strain 

None 

Slight 

Moderate 

Severe 

5.  Difficulty  Focusing 

None 

Slight 

Moderate 

Severe 

6.  Increased  Salivation 

None 

Slight 

Moderate 

Severe 

7.  Sweating 

None 

Slight 

Moderate 

Severe 

8.  Nausea 

None 

Slight 

Moderate 

Severe 

9.  Difficulty  Concentrating 

None 

Slight 

Moderate 

Severe 

10.  Fullness  of  Head 

None 

Slight 

Moderate 

Severe 

1 1.  Blurred  Vision 

None 

Slight 

Moderate 

Severe 

12.  Dizzy  (Eyes  Open) 

None 

Slight 

Moderate 

Severe 

13.  Dizzy  (Eyes  Closed) 

None 

Slight 

Moderate 

Severe 

14.  Vertigo* 

None 

Slight 

Moderate 

Severe 

15.  Stomach  Awareness 

None 

Slight 

Moderate 

Severe 

16.  Burping 

None 

Slight 

Moderate 

Severe 

*Vertigo  is  a  disordered  state  in  which  the  person  or  his/her  surroundings  seem  to  whirl  dizzily: 
giddiness. 
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APPENDIX  C:  PRESENCE  QUESTIONNAIRE 


PRESENCE  QUESTIONNAIRE 
(Witmer  &  Singer,  Vs.  3.0,  Nov.  1994) 

Characterize  your  experience  in  the  environment,  by  marking  an  "X"  in  the  appropriate  box  of 
the  7-point  scale,  in  accordance  with  the  question  content  and  descriptive  labels.  Please  consider 
the  entire  scale  when  making  your  responses,  as  the  intermediate  levels  may  apply.  Answer  the 
questions  independently  in  the  order  that  they  appear.  Do  not  skip  questions  or  return  to  a 
previous  question  to  change  your  answer. 

WITH  REGARD  TO  THE  EXPERIENCED  ENVIRONMENT 

1 .  How  much  were  you  able  to  control  events? 


NOT  AT  ALL  SOMEWHAT  COMPLETELY 

2.  How  responsive  was  the  environment  to  actions  that  you  initiated  (or  performed)? 

I _ I _ I _ I _ I _ I _ I _ I 

NOT  MODERATELY  COMPLETELY 

RESPONSIVE  RESPONSIVE  RESPONSIVE 

3.  How  natural  did  your  interactions  with  the  environment  seem? 

I _ I _ I _ I _ I _ I _ I _ I 

EXTREMELY  BORDERLINE  COMPLETELY 

ARTIFICIAL  NATURAL 

4.  How  much  did  the  visual  aspects  of  the  environment  involve  you? 

I _ I _ I _ I _ I _ I _ I _ I 

NOT  AT  ALL  SOMEWHAT  COMPLETELY 

5.  How  much  did  the  auditory  aspects  of  the  environment  involve  you? 

I _ I _ I _ I _ I _ I _ I _ I 

NOT  AT  ALL  SOMEWHAT  COMPLETELY 

6.  How  natural  was  the  mechanism  which  controlled  movement  through  the  environment? 
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EXTREMELY  BORDERLINE  COMPLETELY 

ARTIFICIAL  NATURAL 

7.  How  compelling  was  your  sense  of  objects  moving  through  space? 


NOT  AT  ALL  MODERATELY  VERY 

COMPELLING  COMPELLING 

8.  How  much  did  your  experiences  in  the  virtual  environment  seem  consistent  with  your  real 
world  experiences? 


NOT  MODERATELY  VERY 

CONSISTENT  CONSISTENT  CONSISTENT 

9.  Were  you  able  to  anticipate  what  would  happen  next  in  response  to  the  actions  that  you 
performed? 


NOT  AT  ALL  SOMEWHAT  COMPLETELY 

10.  How  completely  were  you  able  to  actively  survey  or  search  the  environment  using  vision? 


NOT  AT  ALL  SOMEWHAT  COMPLETELY 

1 1 .  How  well  could  you  identify  sounds? 


NOT  AT  ALL  SOMEWHAT  COMPLETELY 

12.  How  well  could  you  localize  sounds? 


NOT  AT  ALL  SOMEWHAT  COMPLETELY 

1 3.  How  well  could  you  actively  survey  or  search  the  virtual  environment  using  touch? 


NOT  AT  ALL  SOMEWHAT  COMPLETELY 

14.  How  compelling  was  your  sense  of  moving  around  inside  the  virtual  environment? 
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NOT 

MODERATELY 

VERY 

COMPELLING 

COMPELLING 

COMPELLING 

1 5.  How  closely  were 

1  1  .  1 

you  able  to  examine  objects? 

1 

NOT  AT  ALL 

PRETTY 

VERY 

CLOSELY 

CLOSELY 

16.  How  well  could  you  examine  objects  from  multiple 

1  1  1  1  1  1 

viewpoints? 

1  1 

NOT  AT  ALL 

SOMEWHAT 

EXTENSIVELY 

1 7.  How  well  could  you  move  or  manipulate  objects  in  the  virtual  environment? 


NOT  AT  ALL  SOMEWHAT  EXTENSIVELY 

1 8.  How  involved  were  you  in  the  virtual  environment  experience? 


NOT  MILDLY  COMPLETELY 

INVOLVED  INVOLVED  ENGROSSED 

1 9.  How  much  delay  did  you  experience  between  your  actions  and  expected  outcomes? 


NO  DELAYS  MODERATE 

DELAYS 


LONG 

DELAYS 


20.  How  quickly  did  you  adjust  to  the  virtual  environment  experience? 


NOT  AT  ALL  SLOWLY  LESS  THAN 

ONE  MINUTE 
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21 .  How  proficient  in  moving  and  interacting  with  the  virtual  environment  did  you  feel  at  the 
end  of  the  experience? 


NOT  REASONABLY  VERY 

PROFICIENT  PROFICIENT  PROFICIENT 

22.  How  much  did  the  visual  display  quality  interfere  or  distract  you  from  performing  assigned 
tasks  or  required  activities? 


NOT  AT  ALL  INTERFERED  PREVENTED 

SOMEWHAT  TASK  PERFORMANCE 

23.  How  much  did  the  control  devices  interfere  with  the  performance  of  assigned  tasks  or  with 
other  activities? 


NOT  AT  ALL  INTERFERED  INTERFERED 

SOMEWHAT  GREATLY 

24.  How  well  could  you  concentrate  on  the  assigned  tasks  or  required  activities  rather  than  on 
the  mechanisms  used  to  perform  those  tasks  or  activities? 


NOT  AT  ALL  SOMEWHAT  COMPLETELY 

25.  How  completely  were  your  senses  engaged  in  this  experience? 


NOT  MILDLY  COMPLETELY 

ENGAGED  ENGAGED  ENGAGED 

26.  To  what  extent  did  events  occurring  outside  the  virtual  environment  distract  from  your 
experience  in  the  virtual  environment? 

I _ I _ I _ I _ I _ I _ I _ I 

NOT  AT  ALL  MODERATELY  VERY  MUCH 

27.  Overall,  how  much  did  you  focus  on  using  the  display  and  control  devices  instead  of  the 
virtual  experience  and  experimental  tasks? 

I _ I _ I _ I _ I _ I _ I _ I 

NOT  AT  ALL  SOMEWHAT  VERY  MUCH 

28.  Were  you  involved  in  the  experimental  task  to  the  extent  that  you  lost  track  of  time? 
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NOT  AT  ALL  SOMEWHAT  COMPLETELY 

29.  How  easy  was  it  to  identify  objects  through  physical  interaction;  like  touching  an  object, 
walking  over  a  surface,  or  bumping  into  a  wall  or  object? 


IMPOSSIBLE  MODERATELY  VERY  EASY 

DIFFICULT 

30.  Were  there  moments  during  the  virtual  environment  experience  when  you  felt  completely 
focused  on  the  task  or  environment? 


NONE  OCCASIONALLY  FREQUENTLY 

3 1 .  How  easily  did  you  adjust  to  the  control  devices  used  to  interact  with  the  virtual 
environment? 


DIFFICULT  MODERATE  EASILY 

32.  Was  the  information  provided  through  different  senses  in  the  virtual  environment  (e.g., 
vision,  hearing,  touch)  consistent? 


NOT 

CONSISTENT 


SOMEWHAT 

CONSISTENT 


VERY 

CONSISTENT 
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